Course Instructor

  • Curtis Phills, PhD
  • Office: 51/3429
  • Email: curtis.phills@unf.edu
  • Office Hours: T/R noon to 3 pm
  • Course Website: Canvas

Expectations

  • Class begins and ends on time
  • Listen to each other with respect and trust
  • Invest in your own learning
  • Participate actively in your learning

Today you will have the opportunity to:

  • Identify some of the tools psychologists use to conduct research and communicate findings
  • Define what you want to learn

Agenda

  • Class structure
  • What makes a study convincing?
  • The Scientific Method
  • Transparent Science

Class Format

  • Each class will usually consist of:
    • lecture
    • discussion
    • group/individual work
    • maybe a video

Course Textbook

Goals of this course

  • At the end of this course you will be able to:
    • Critically evaluate social psychological research.
    • Integrate your unique hypotheses with relevant published research.
    • Design and build an experiment using modern tools.
    • Recruit participants while protecting their rights as human subjects.
    • Use data analysis software to evaluate social psychological research outcomes.
    • Report your research results in an oral presentation and in written form in accordance with APA guidelines.

Assignments

Assessment Group/Individual Points
Class Notebooks Individual 60
Online Homework Individual 50
Review Exam Individual 50
CITI Certification Individual 50
1st Qualtrics Individual 10
Project Prep Individual 40
Project Prep Group 110
Final Paper Individual 100
Final Presentation Individual 50

Grading Scheme

Letter Grade Grading Scale (%)
A 90-100
B+ 87-89
B 80-86
C+ 77-79
C 70-76
D 60-69
F 00-59

What makes a study convincing?

  • low p-value
  • lots of participants
  • probability sample
  • preregistration
  • replication
  • can check the data myself

Write down everytime someone is surprised

When are we surprised?

  • When an assumption is violated

What is a p-value?

  • A p-value represents how surprising the data is, assuming the null hypothesis is true

What is a p-value?

  • A p-value represents how surprising the data is, assuming the null hypothesis is true
  • Null hypothesis: Harry is not a wizard
  • Alternative hypothesis: Harry is a wizard
  • Data: Hagrid told Harry he is a wizard.
  • Should we reject the null hypothesis? How surprising is this data, assuming Harry is not a wizard.

What is a p-value?

  • A p-value represents how surprising the data is, assuming the null hypothesis is true
  • A p-value is significant when it is less than .05
  • In your mind, switch the word ‘significant’ to ‘surprising, assuming the null hypothesis is true’
  • A p-value is the probability of seeing data as or more extreme than the current data, assuming the null hypothesis is true.

My p-value is less than .05

  • Does this mean there is a 95% probability the alternative hypothesis is true?

What is a p-value?

  • A p-value represents how surprising the data is, assuming the null hypothesis is true
  • Scientists use an alpha level to decide whether to reject the null hypothesis. It is usually set to p < .05.
  • This does not mean there is a 95% chance the alternative hypothesis is correct.
  • In fact, we never know if we were correct in rejecting the null hypothesis.
  • We just know that in the long run if we set our alpha level to p < .05, we’ll never be wrong more than 5% of the time.

Flip a coin!

  • Or ask your phone to

What is an effect size?

  • The magnitude of an effect

Are women more extraverted than men?

Do women report more negative affect than men?

Are women more agreeable than men?

Think-Pair-Share

  • Are women better coders than men?
  • Break into groups of 4-6
  • Write your names and n-numbers on a piece of paper
  • Women are better coders than men: p < .05, d = .04
  • Imagine a man and a woman have applied to be a coder for your business. Would you automatically hire the woman? How much more likely would you want a randomly selected woman to be better at coding before you would use gender as a basis for hiring a coder?
  • Once your group has reached consensus, hand in your answers to the three questions.

What is statistical power?

  • Assuming the null hypothesis is false, what is the probability the p-value will be less than .05?

What is statistical power?

  • Assuming the null hypothesis is false, what is the probability the p-value will be less than .05?
  • related to effect size and sample size
  • Statistical power increases when you have a larger sample
  • Statistical power increases when you have a larger effect
  • Rule of thumb is to aim for 80% power

P-values and sample sizes

  • Group A: M = 100
  • Group B: M = 106
  • (there is a real difference between these 2 groups)
  • 20 participants in each group
  • What is the p-value?

P-values and sample sizes

  • Group A: M = 100
  • Group B: M = 106
  • (there is a real difference between these 2 groups)
  • 30 participants in each group
  • What is the p-value?

P-values and sample sizes

  • Group A: M = 100
  • Group B: M = 106
  • (there is a real difference between these 2 groups)
  • 50 participants in each group
  • What is the p-value?

P-values and sample sizes

  • Group A: M = 100
  • Group B: M = 106
  • (there is a real difference between these 2 groups)
  • 100 participants in each group
  • What is the p-value?

What does a p-value tell us?

Optional Stopping

  • This is a how a p-value changes as you add more participants.

  • This represents the case where the null hypothesis is true.

What does a p-value tell us?

  • To answer this we need to know:
    • How many tests conducted
    • How many participants

The Scientific Method

Design
Collect
Analyze
Analyze
Analyze
Publish!

The Scientific Method

Design
Collect
Analyze
Design
Collect
Analyze
Design
Collect
Analyze
Publish!

QRPs, HARKING, and P-Hacking

QRP
Questionable Research Practice
HARKING
Hypothesizing after results known
p-hacking
another phrase to describe a questionable research practice but usually with a focus on obtaining a significant p-value

Preregistration

Preregistration

  • Register details of a study beforehand so you don’t exercise researcher degrees of freedom later
  • Design
  • Sample size/Stopping rule
  • Analysis Plan and exclusion rules

External Validity

  • The extent to which results of a study can be generalized to other populations and settings
  • Exact replication: generalize to other labs
  • Conceptual replication: generalize to other operationalizations of variables
  • Other populations
  • Other experimenters
  • Replication plus extension: researchers replicate original study but add variables to test additional questions

Replication vs. Reproduction

Replication
If I re-run your experiment, do I get the same results?
  • About 35% of Social Psychology experiments replicate
Reproduction
If I re-analyze your data, do I get the same results?

In the journal cognition:

  • About 78% of data is available to be re-analyzed
  • About 60% of data is actually usable
  • Of 35 articles selected for a test:
    • 11 could be reproduced
    • 11 more could be reproduced with help from the authors
    • 13 articles not be reproduced even with author help

Why is replication important to the scientific method?

The Scientific Method

Design
Collect
Analyze
Replicate

Ego Depletion

  • Willpower is like a muscle. It gets tired as you use it. So you have less willpower and doing a task that requires willpower.
  • Haggar et al., 2010: Meta-analysis of 83 studies finds evidence for ego depletion (d = .63—medium)
  • Carter & McCullough (2014): Another meta-analysis but correcting for small study effects (d = 0)
  • Haggar et al., 2016: Many labs study (23 labs), 2141 participants. Ego depletion is a small effect (d ~= 0).

Open Science

Next week